Repository navigation
feat(ci): verify agent trust and preserve generated API contracts - #384
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8c24582a6e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 03019365fa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
dodeja
changed the title
dodeja
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8594722586
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4bcea9bab7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0f5312ab72
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2245417b57
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
2245417 to
93de765
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4241fb6b4e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ]), | ||
| { | ||
| name: 'OpenAPI SDK type generation', | ||
| command: ['node', 'bin/check-openapi-types.mjs'], |
There was a problem hiding this comment.
When an OpenAPI filter description or validation rule changes and the canonical SDK generator is run, this step verifies only the SDK artifacts; no agent-verify step regenerates and compares packages/mcp/src/generated/list-filter-schemas.ts. The workspace build and tests consume that stale file, so the required handoff gate can pass while the MCP tool schemas still expose outdated metadata or validation. Run the MCP filter generator in isolation and compare its output as part of this gate.
AGENTS.md reference: AGENTS.md:L51-L54
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in f1b3629. bin/check-openapi-types.mjs now also runs packages/mcp/scripts/generate-list-filter-schemas.mjs in its temporary checkout and diffs packages/mcp/src/generated/list-filter-schemas.ts. agent-verify runs that check, and so does the new CI step (npm run check:agent-trust). bin/check-openapi-types.test.mjs has a new case, rejects drift in packages/mcp/src/generated/list-filter-schemas.ts. It fails against the old checker and passes with the fix.
| if (options.incremental || options.composite) | ||
| options.tsBuildInfoFile = join(temporary, 'build.tsbuildinfo'); |
There was a problem hiding this comment.
When the SDK enables incremental or composite, this assigns tsBuildInfoFile under the temporary directory, but program.emit passes that .tsbuildinfo file through the same write callback; the callback then rejects it for being outside the configured dist directory. Consequently, agent-verify fails for either otherwise-valid compiler option despite this branch attempting to support them. Handle the build-info output separately from package artifacts.
AGENTS.md reference: AGENTS.md:L82-L82
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in f1b3629. You were right. The new tests accepts incremental SDK builds whose build info stays out of dist and accepts composite SDK builds whose build info stays out of dist failed with SDK compiler output escaped configured dist: .../build.tsbuildinfo. The emit callback now skips the configured tsBuildInfoFile. The SDK tsconfig sets neither option today, so CI was not affected.
`npm run agent-verify` is the local completion gate. It runs the agent trust guards, the Vite+ and TypeScript checks, every workspace build and test, the OpenAPI generation drift check, and the offline MCP protocol smoke. - bin/check-agent-trust.mjs keeps skills/agent-trust/feature-map.json in sync with the MCP tools registered in server.ts, the SDK package exports, and the MCP and SDK docs navigation. It also checks the vite-plus toolchain pins, the ordered workspace build, and that no workspace has its own Vite config. - bin/check-openapi-types.mjs regenerates the SDK types, the SDK filter reference, and the MCP filter schemas from docs/openapi.json in a temporary checkout and fails on any drift. - The feature map covers main as it is now: 14 tools (whoami included) and the Gemini and Antigravity setup guides. - The guard skips the TypeScript build-info file when the SDK compiles incrementally or as a composite project. Before this, that file failed the dist check. - CI runs the guards and the drift check as a new step in the toolchain job. The existing check, test, and build steps stay as they are. Rebuilt on current main from #384's branch (head 4241fb6). The /containers/{id}/route OpenAPI path, its schemas, and its reference page are left out. Main removed those docs in 2792425. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type-checks small consumer programs against the generated OpenAPI types. A container payload with type "container" compiles. An account discriminant is rejected with TS2322. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#421 edited the title and description in the generated list-filters.mdx but not in the script that generates it. The next regeneration (#418) put the old metadata back. This change moves #421's metadata into generate-filter-types.mjs and regenerates the page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4241fb6 to
c6a11cb
Compare
) * feat(ci): reconcile advisory PR trust policy from current evidence Add an advisory trust-policy controller that classifies changed paths by risk, checks current-head human writer approvals and exact-attempt CI evidence, and writes risk labels and summaries. It never authorizes a merge; native branch protection stays the merge authority. - trust-policy.yml, shadow-risk.yml and trust-review-signal.yml run trusted default-branch code only. - scripts/ci holds the classifier, the shadow scorer, the read-only branch-protection checker and the reconciler, with regression tests. - CODEOWNERS names owners for every high-risk path. - npm test and agent-verify now run the policy tests. Rebuilt from the reviewed history at 6b5f819 on top of the rebuilt agent-trust base; the Postman bot commit is not carried over. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: add the trust-policy installation guide and GitHub trial Keep the maintainer guides with the agent-trust skill and link them from the paved path, so maintainers run the GitHub verification trial before they change merge protection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ci): own every agent-trust verifier and drop moved guide paths bin/check-openapi-types.test.mjs was classified medium risk and had no CODEOWNERS entry, unlike the other verifier tests. The new test derives the verifier list from bin/, so a future verifier cannot miss either. The docs/risk-gates*.md entries pointed at guides that now live under skills/agent-trust/, which the existing prefix rules already cover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: Auto-generate Postman collection from openapi.json [skip ci] --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
The generator assigns random request IDs and enum example values, so each rerun rewrites about 640 lines of the collection from an unchanged docs/openapi.json. This branch does not change the spec, so it carries main's collection and the PR diff has no generated churn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Reconciled with main. History was not rewritten.
CI on Locally on the merged tree, The route-doc scope is unchanged. This branch adds no API reference page and does not change Still open for a human:
🤖 Generated with Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ee2b796004
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| receiver && | ||
| ts.isIdentifier(receiver) && | ||
| receiver.text === 'server' && | ||
| method === 'registerTool' | ||
| ) { |
There was a problem hiding this comment.
When a tool is added through an alias such as const toolServer = server; toolServer.registerTool('new_tool', ...), this predicate ignores the registration because it compares the receiver's text only to server. After updating the runtime tool-count and behavior tests, the stale feature map can therefore still pass verification. The replacement AST guard is fresh evidence that the earlier registration-inventory issue remains for aliased receivers; resolve the receiver symbol or explicitly reject unsupported registerTool call forms.
AGENTS.md reference: AGENTS.md:L79-L81
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in cab4117. The inventory no longer looks at the receiver name. Every <receiver>.registerTool(...) or <receiver>['registerTool'](...) call in server.ts counts as a registration. Any other use of registerTool fails with the source line (Unsupported MCP registerTool form at packages/mcp/src/server.ts:N), so destructuring, .bind, or a key held in a variable cannot hide a tool.
New tests in bin/check-agent-trust.test.mjs run the real guard CLI against a temp repository:
registration through an alias is held to the feature mapregistration through a reassigned receiver is held to the feature maprejects the unsupported registration formforconst { registerTool } = server,server.registerTool.bind(server), andserver[method]withmethod = 'registerTool'
With the old checker restored, all 5 fail and the guard reports PASS for each unmapped tool. With the fix, all 57 agent-trust and OpenAPI-type tests pass and node bin/check-agent-trust.mjs still passes on the real server.ts. A built-up key such as server['register' + 'Tool'] is still not detected. The old checker missed it too.
The registration inventory only counted calls whose receiver was the identifier `server`, so `const alias = server; alias.registerTool(...)` or a reassigned receiver registered a tool the feature map never saw. Every `<receiver>.registerTool(...)` call in server.ts now counts as a registration whatever the receiver is. Any other use of `registerTool` (destructuring, `.bind`, a computed key held in a variable) fails with the source line, so a registration cannot hide behind an unsupported form. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Receipts for head cab4117 (
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cab4117701
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const artifacts = [ | ||
| 'sdks/typescript-sdk/src/generated/terminal49.ts', | ||
| 'sdks/typescript-sdk/src/generated/list-filters.ts', | ||
| 'docs/api-docs/api-reference/list-filters.mdx', | ||
| 'packages/mcp/src/generated/list-filter-schemas.ts', |
There was a problem hiding this comment.
When docs/openapi.json changes and the SDK/MCP generators are run, this artifact list still omits Terminal49-API.postman_collection.json, so both check:agent-trust and the required CI job can pass with a stale public collection. The existing Postman workflow runs only after a push, so it does not validate the reviewed commit; generate the collection in isolation and compare it here with the other OpenAPI-derived outputs.
AGENTS.md reference: AGENTS.md:L51-L54
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not changing this. The Postman generator is not deterministic, so a byte-for-byte drift check against it would fail on every run.
Evidence. I ran the pinned generator from generate_postman.yml (openapi-to-postmanv2@6.3.0, -p -O folderStrategy=Tags,enableOptionalParameters=false) twice on this branch's docs/openapi.json:
- The two outputs have different hashes (
90e6cc17…ande4b10089…). Neither matches the committed collection (10f60934…). diff run1.json run2.jsonshows 1286 changed lines from one identical input.- The changes are random request
idUUIDs (e106f9d2-…vsca655926-…) and randomly picked enum example values ("off_dock"vs"delivered").
Normalizing the output would mean dropping every id and every enum example. What remained would not tell a reviewer whether the collection is stale.
The collection is downstream output, not reviewed source. AGENTS.md lists it as generated, not to be hand-edited. generate_postman.yml regenerates and commits it on every push that touches docs/openapi.json, on any branch, PR branches included. This branch already has one of those bot commits (880cfaf). The gate keeps checking the deterministic OpenAPI outputs (SDK types, SDK filters, the filters MDX page, and the MCP filter schemas).
| globSync( | ||
| readJson('package.json').workspaces.map( | ||
| (workspace) => `${workspace}/package.json`, | ||
| ), | ||
| { cwd: root }, |
There was a problem hiding this comment.
When the root workspace configuration uses a recursive pattern such as sdks/**, appending /package.json and passing it directly to globSync also matches installed dependencies—for example, this checkout contains sdks/typescript-sdk/node_modules/openapi-fetch/package.json. Those packages are not npm workspaces, but the guard then requires their toolchain pins and root build entries, causing check:agent-trust to fail after a legitimate workspace-pattern change. The new glob-based implementation and the existing nested dependency manifests are fresh evidence beyond the earlier hard-coded-inventory issue; resolve manifests with npm workspace semantics or exclude node_modules.
AGENTS.md reference: AGENTS.md:L73-L73
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 5727805. You were right: globSync('sdks/**/package.json') returns sdks/typescript-sdk/node_modules/*/package.json too. workspaceManifests() now passes exclude: ['**/node_modules/**']. That is the same ignore npm applies when it maps workspaces (@npmcli/map-workspaces adds '**/node_modules/**' to its glob ignore list).
New test in bin/check-agent-trust.test.mjs, recursive workspace globs skip installed dependency manifests like npm. It sets the root workspaces to ['packages/*', 'sdks/**'], writes sdks/typescript-sdk/node_modules/openapi-fetch/package.json, and runs the real guard CLI against a temp repository.
- With the old guard restored, it fails with
FAIL agent trust guards: Cannot read properties of undefined (reading 'vite-plus'). - With the fix, it passes.
npm run test:agent-trustpasses 58/58 andnpm run check:agent-trustreports PASS on the real repo.
Negated workspace patterns (!path) are still not supported. The root manifest uses none today.
The toolchain guard appended /package.json to each root workspace pattern and globbed it as-is, so a recursive pattern such as sdks/** also matched installed dependency manifests (sdks/typescript-sdk/node_modules/*/package.json) and failed on their missing toolchain pins. npm ignores **/node_modules/** when it maps workspaces; the guard now does the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The advisory sweep awaited every PR in sequence, so one stalled GitHub request could hold the job until its 20-minute timeout and starve every later PR. Each PR now runs under an AbortController deadline. When it fires, the PR is reported as failed, its in-flight direct request is aborted, any later label write for that PR is rejected before it is sent, and the sweep moves on to the next PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A PR that hit its deadline could still finish later (a stalled paginated call is not aborted) and append its report to the shared job summary, next to the failure that said it timed out. Each PR now writes into its own buffer, and only a PR that finishes within its deadline flushes it to the job summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3ba50e5104
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| name: 'Policy and workspace tests', | ||
| command: ['npm', 'test'], |
There was a problem hiding this comment.
When an agent changes a coverage generator or its inputs without regenerating the published coverage artifacts, the required full npm run agent-verify can still pass: this step invokes npm test, whose root script does not include the separate test:coverage command, and no later step runs it either. Although the dedicated CI coverage job will catch this after handoff, the documented completion gate gives a false local success; include npm run test:coverage in the non-quick sequence.
AGENTS.md reference: AGENTS.md:L82-L82
Useful? React with 👍 / 👎.
| name: 'OpenAPI SDK and MCP generation', | ||
| command: ['node', 'bin/check-openapi-types.mjs'], |
There was a problem hiding this comment.
When public SDK declarations or JSDoc change without running npm run sdk:docs, every agent-verify step can succeed because this generation check covers only the OpenAPI-derived files and nothing regenerates or compares docs/sdk/reference/**. The separate sdk-tooling CI job catches the drift later, but the required handoff gate incorrectly reports success with stale generated public documentation; run the canonical SDK docs generator in isolation and compare its outputs here.
AGENTS.md reference: AGENTS.md:L51-L55
Useful? React with 👍 / 👎.
|
Receipts for this pass. Head moved cab4117 -> 3ba50e5 -> df81cbd. All three commits are signed fast-forward pushes. Nothing was merged, closed, resolved, retargeted or force-pushed. The route-doc scope is unchanged. Workspace globs (Codex P2, bin/check-agent-trust.mjs:328). Fixed in 5727805. The proof is in the thread: #384 (comment) Postman drift (Codex P2, bin/check-openapi-types.mjs:23). Not changed. The pinned generator is not deterministic: two runs on the same spec differ by 1286 lines. The evidence is in the thread: #384 (comment) Per-PR deadline (Codex review finding, scripts/ci/reconcile-trust.mjs:345-353; no inline thread). Fixed in 3ba50e5 and df81cbd.
Local results on df81cbd. Independent review at df81cbd. Opus 5.5 and GPT-6 astra (xhigh) both returned no blockers. Opus reverted each fix and confirmed every new test fails without it. Remaining P3s are recorded, not fixed:
CI on df81cbd. 22 checks pass. Not merge-ready. These gates remain: |
Brings in trade intelligence (#415) and the CI trust gates (#384). The seven trade intelligence tools move into the tool registry under a new `trade` toolset. It is on by default and stays on for the shared env token, because every tool in it is read-only, so the default tools/list keeps everything main serves today. Names, titles, descriptions, schemas and annotations are unchanged. The per-account entitlement is still checked when a tool runs, not when it is registered. The server instructions keep main's trade guidance as their own paragraph, dropped when the session does not serve the trade toolset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
1 participant
Adds a repository verifier for agent-authored changes, plus the advisory trust and risk policy from #383, which was squash-merged into this branch.
Verifier.
npm run check:agent-trustruns in CI beforevp run check. It checks thatskills/agent-trust/feature-map.jsonmatches the 14 registered MCP tools and their implementation paths, the public SDK exports, and every page in the SDK/MCP docs navigation. EveryregisterToolcall inserver.tscounts as a registration, whatever its receiver. A registration form the guard cannot read fails with its source line. The verifier also checks workspace toolchain ownership. It finds workspaces from the rootworkspacesglobs and skipsnode_modules, as npm does. Last, it checks that the generated SDK and MCP OpenAPI artifacts matchdocs/openapi.json.npm run agent-verifyruns the full local sequence.The Postman collection is not drift-checked. Its generator is not deterministic: two runs on the same spec differ by about 1,300 lines of random UUIDs and enum examples.
generate_postman.ymlregenerates the collection on every push that changesdocs/openapi.json.Generated contract test.
sdks/typescript-sdk/src/generated-contract.test.tscompiles consumers against the generated types and pins the container resource discriminant.Advisory trust policy (from #383). This adds
.github/workflows/trust-policy.yml,shadow-risk.yml,trust-review-signal.ymland.github/CODEOWNERS. It also adds thescripts/ci/risk classifier, config reader, trust reconciler and shadow scorer, tested undernpm run test:risk-policy. The reconciler gives each PR a 60s deadline. A stalled GitHub request fails that PR, aborts its pending label writes, keeps its report out of the job summary, and lets the sweep move on to the next PR.Docs. The list filter reference keeps the SEO title and description from #421 in its generator. Agent guidance lives in
skills/agent-trust/,.cursor/rules/and a shortAGENTS.mdsection.This PR does not change
docs/openapi.json, add an API reference page, or change any MCP tool. The generated Postman collection matches main.Verification. On head df81cbd, local runs passed
npm run test:agent-trust(58),npm run test:risk-policy(49),npm run check:agent-trustandvp check. The reconciler deadline was also tested against the real@actions/githubv6 client and a local server that never answers. In CI, every check passes except the authenticated "MCP preview 2026-07-28" gate. That gate fails because the sharedMCP_EVAL_TOKENcredential is rejected, not because of this code.Remaining gates.
pull_request_targetand schedule), so the reconciler has not run live. Its first live sweep happens after merge.Before merging. Three commit messages on this branch contain
[skip ci]: Postman bot commits 8b41bb4 and 880cfaf, and a bullet in the body of the #383 squash e909f5d. The repository builds squash messages from commit messages. Remove those lines from the squash message, or main skips CI, Postman generation and release-please.🤖 Generated with Claude Code